Computerized method and apparatus for automated distributed ingest and presentation of multi-domain metrics

ABSTRACT

A computerized search method includes receiving first input designating a first location of a first analytic. The method includes, in response to a scheduling event, obtaining a first document from the first location, identifying a first predefined label within the first document, obtaining first and second data associated with the first predefined label, storing the first datum into a value index as a current value of the first analytic, and storing the second datum into a text index as a textual description of the first analytic. The method includes presenting a search interface and, in response to receiving a search query from a user: identifying a set of result analytics relevant to the search query based on the text index and presenting, for each of the result analytics, a textual description of the analytic from the text index and a most recent value of the analytic from the value index.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/947,064, filed on Apr. 6, 2018; the entire disclosure of which isincorporated herein by reference.

FIELD

The present disclosure relates to computerized search engines, and moreparticularly to natural language text-based computerized search engines.

BACKGROUND

Traditional search engines crawl a set of documents, such as web pages,and attempt to infer semantic meaning from contents and metadata ofthose documents. Traditional search engines, however, are not adept atidentifying which portions of these documents are most important,especially when the documents are data-intensive. For example, while atraditional search engine may be able to index text found within cellsof a spreadsheet, the search engine may not have any understanding ofthe meaning or importance of data within the spreadsheet.

The background description provided here is for the purpose of generallypresenting the context of the disclosure. Work of the presently namedinventors, to the extent it is described in this background section, aswell as aspects of the description that may not otherwise qualify asprior art at the time of filing, are neither expressly nor impliedlyadmitted as prior art against the present disclosure.

SUMMARY

A computerized search method includes specifying a first predefinedlabel for use with the computerized search method. The method includesreceiving first input designating a first location of a first analytic.The method includes, in response to a scheduling event, (i) obtaining afirst document from the first location, (ii) identifying the firstpredefined label within the first document, (iii) obtaining a firstdatum associated with the first predefined label from the firstdocument, (iv) storing the first datum into a value index as a currentvalue of the first analytic, (v) obtaining a second datum associatedwith the first predefined label from the first document, and (vi)storing the second datum into a text index as a textual description ofthe first analytic. The method includes presenting a search interface.The method includes, in response to receiving a search query from a uservia the search interface: identifying a set of result analytics relevantto the search query based on the text index and, using the searchinterface, presenting, for each analytic of the set of result analytics,a textual description of the analytic from the text index and a mostrecent value of the analytic from the value index.

In other features, the method includes receiving second inputdesignating a second location of a second analytic and position data andstoring the second location and the position data. The method includes,in response to a second scheduling event, obtaining a second documentfrom the second location, extracting a value from within the seconddocument at a position specified by the position data, and storing thevalue into the value index as a current value of the second analytic.

In other features, the method includes receiving second inputdesignating a second location of a second analytic. The method includes,in response to a second scheduling event, obtaining a second documentfrom the second location, extracting a set of values from a set ofpredefined positions within the second document, obtaining a thirddocument from a third location, extracting a third datum from the thirddocument at a position specified by the set of values, and storing thethird datum into the value index as a current value of the secondanalytic.

In other features, the method includes extracting the third locationfrom a predefined position within the second document. In otherfeatures, the method includes, in response to the second schedulingevent, extracting a text value from a predefined position within thesecond document and storing the text value into the text index as atextual description of the second analytic.

In other features, the first document is a spreadsheet workbook. Aworksheet of the workbook is selected in response to a name of theselected worksheet matching the first predefined label. The first datumis obtained from a first predetermined row and a first predeterminedcolumn of the selected worksheet. The second datum is obtained from thefirst predetermined row and a second predetermined column of theselected worksheet. In other features, the first location is a uniformresource locator (URL).

In other features, the method includes, in response to the schedulingevent, determining whether a modification date is available for thefirst document. The method includes, in response to determining that themodification date is available for the first document, obtaining themodification date, comparing the modification date to a most recentacquisition date of the first analytic, and obtaining the first documentonly if the modification date is later than the most recent acquisitiondate.

In other features, the first document is a file located in a file storecontrolled by an operating system. The modification date is obtained bysending a request to the operating system. In other features, the methodincludes receiving second input specifying a refresh schedule of thefirst analytic, storing information specifying the refresh scheduleassociated with the first analytic, and generating the scheduling eventbased on the refresh schedule.

In other features, the method includes, in response to obtaining thefirst document from the first location, extracting a list associatedwith the first predefined label from the first document. The listidentifies a set of analytics including the first analytic. The methodincludes, for each analytic of the set of analytics, storing anassociated value into the value index as a current value of the analyticand storing an associated textual description into the text index as atextual description of the analytic.

In other features, the first document includes tabular data. Each of theset of analytics corresponds to a different row of the tabular data.Textual descriptions for the set of analytics are located in a firstcolumn of the tabular data. Values for the set of analytics are locatedin a second column of the tabular data. In other features, the methodincludes, in response to obtaining the first document, acquiring, from apredefined position in the first document with respect to the firstpredefined label, identification information for an author of the firstanalytic. The method includes selectively displaying the identificationinformation to the user when the first analytic is included in the setof result analytics.

In other features, the method includes, in response to obtaining thefirst document, acquiring, from a second predefined position in thefirst document with respect to the first predefined label, contactinformation for the author and selectively displaying the contactinformation to the user when the first analytic is included in the setof result analytics. In other features, the method includes, in responseto obtaining the first document, acquiring, from a predefined positionin the first document with respect to the first predefined label, anaccess uniform resource locator (URL) for a source of data underlyingthe first analytic. The method includes selectively displaying theaccess URL to the user when the first analytic is included in the set ofresult analytics.

In other features, the method includes, in response to obtaining thefirst document, acquiring, from a predefined position in the firstdocument with respect to the first predefined label, a modificationtimestamp for the first analytic and controlling generation of a futurescheduling event based on the modification timestamp. In other features,the method includes, in response to obtaining the first document,acquiring, from a predefined position in the first document with respectto the first predefined label, a name of the first analytic. The methodincludes storing the name into the text index for the first analytic. Inother features, the method includes operating a web server configured toreceive the first input and present the search interface.

A system includes at least one processor and a computer-readable mediumconfigured to store instructions for execution by the at least oneprocessor. The instructions include specifying a first predefined labelfor use with the computerized search method. The instructions includereceiving first input designating a first location of a first analytic.The instructions include, in response to a scheduling event: (i)obtaining a first document from the first location, (ii) identifying thefirst predefined label within the first document, (iii) obtaining afirst datum associated with the first predefined label from the firstdocument, (iv) storing the first datum into a value index as a currentvalue of the first analytic, (v) obtaining a second datum associatedwith the first predefined label from the first document, and (vi)storing the second datum into a text index as a textual description ofthe first analytic. The instructions include presenting a searchinterface. The instructions include, in response to receiving a searchquery from a user via the search interface, identifying a set of resultanalytics relevant to the search query based on the text index and,using the search interface, presenting, for each analytic of the set ofresult analytics, a textual description of the analytic from the textindex and a most recent value of the analytic from the value index.

A non-transitory computer-readable medium stores processor-executableinstructions including specifying a first predefined label for use withthe computerized search method. The instructions include receiving firstinput designating a first location of a first analytic. The instructionsinclude, in response to a scheduling event: (i) obtaining a firstdocument from the first location, (ii) identifying the first predefinedlabel within the first document, (iii) obtaining a first datumassociated with the first predefined label from the first document, (iv)storing the first datum into a value index as a current value of thefirst analytic, (v) obtaining a second datum associated with the firstpredefined label from the first document, and (vi) storing the seconddatum into a text index as a textual description of the first analytic.The instructions include presenting a search interface. The instructionsinclude, in response to receiving a search query from a user via thesearch interface, identifying a set of result analytics relevant to thesearch query based on the text index and, using the search interface,presenting, for each analytic of the set of result analytics, a textualdescription of the analytic from the text index and a most recent valueof the analytic from the value index.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description, the claims, and the drawings.The detailed description and specific examples are intended for purposesof illustration only and are not intended to limit the scope of thedisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from thedetailed description and the accompanying drawings.

FIG. 1 is a functional block diagram of an example system including ahigh-volume pharmacy.

FIG. 2 is a functional block diagram of an example pharmacy fulfillmentdevice, which may be deployed within the system of FIG. 1.

FIG. 3 is a functional block diagram of an example order processingdevice, which may be deployed within the system of FIG. 1.

FIG. 4 is a functional block diagram of an example data analysisenvironment including a search system according to the principles of thepresent disclosure.

FIG. 5 is a functional block diagram of an example implementation of asearch engine.

FIG. 6 is a functional block diagram of an example implementation of aharvesting system.

FIG. 7 is a flowchart of example operation of the administrativefrontend for the harvesting system.

FIG. 8 is a flowchart of example operation of the harvesting system.

In the drawings, reference numbers may be reused to identify similarand/or identical elements.

DETAILED DESCRIPTION Introduction

In data-rich environments, analysts prepare analyses for use by variousend users by identifying and gathering data and generating metrics (alsocalled analytics) based on the data. For example, engineers may refinemanufacturing processes based on quality or timing metrics of amanufacturing system. Business managers may make investment decisionsbased on certain metrics. Users in sales and marketing may target theirefforts based on metrics. Legal and compliance users may rely on metricsto certify statements to a regulator, licensor, etc. Certain metrics maybe difficult to create, relying on data from a variety of sources andsophisticated classification and processing techniques. Further, thesemetrics may change over time. In many instances, these metrics mayexplicitly be time-based, such as a measure of manufacturing output in aprior week.

Even though an analyst may have prepared a first metric, other analysts(and especially non-analyst end users) may not be aware of the metric orrecall that the first metric exists. The users may then request that ananalyst recreate the first metric. In some environments, even theoriginal analyst may have forgotten about the creation of the firstmetric. Traditionally, the output of analysts can be distributed tovarious end users via email distribution lists, which may announce thecreation of new metrics and may periodically provide reports includingthe metrics. When a user needs a metric, the user would traditionallyconsult their own memory and then review their email system; then, usersmight need to discuss the availability of metrics among themselves andwith analysts.

The present disclosure describes a technological solution to theinefficient and uncoordinated approach to analysis distribution anddiscovery. A publisher, who may be the analyst themselves, creates acomputer-readable description of a metric that may be of interest and isnot readily available elsewhere. This computer-readable description isprepared according to the principles of the present disclosure andspecifies information about the metric, including a textual descriptionof the metric, a source from which the metric can be obtained, and theschedule on which the metric is updated.

This computer-readable description is provided to a harvesting systemthat will, according to the specified schedule, obtain the most recentvalue for that metric and store that value in a search engine. Thesearch engine is made accessible to users, who can then query the searchengine to find answers to their questions. These answers take the formof the metrics obtained by the harvesting system.

For users who are aware that a certain metric exists, the search engineprovides rapid retrieval and display of the answer without a manualprocess of consulting colleagues or analysts. In other words, becausethe search engine obtains and is configured to present the value of themetric, the value can be immediately displayed to the user, without theuser having to navigate to the source of the analysis. Further, there isno delay caused by opening particular software (such as a spreadsheetprogram) or retrieving a file (such as a large spreadsheet) over anetwork connection. However, for additional metrics or backgroundinformation, the search engine may offer a link for the user to easilyreach the source of the analysis. In various situations, metrics may bedisplayed to users who don't even have installed software that iscapable of accessing the underlying data source. For example, a businessmanager may be able to see a value for a metric even though the businessmanager does not have a database program capable of performing a SQLquery.

Meanwhile, a user who is unaware of a metric can discover the existenceof the metric through the search engine. In addition, the search enginemay accumulate historical values of the metrics, creating an easilyaccessible historical record of metrics that may not even be availablefrom the analysis source. In contrast to traditional search systems, thesearch engine of the present disclosure does not attempt to parse thetext from the content and metadata of a document and infer semanticmeaning. Instead, the harvesting system relies on the computer-readabledescription of the metric to describe and directly access the metricitself from the analysis source.

Although traditional search systems may be able to present excerpts froma document, such as excerpts including a specified text keyword, thereis no guarantee in data-intensive analytical sources that any givenmetric will be within close proximity to descriptive text correspondingto the user's query, or that the proximate text will be adequatelydescriptive to anyone other than the originating analyst. Analysts mayuse complex data structures in spreadsheets, databases, and businessintelligence (BI) software to aggregate, process, and condense data. Asa result, the metric may correspond to a certain cell within aspreadsheet or a certain field within a certain record of a database.The publisher of the metric can indicate to the harvesting system thisprecise location, which a traditional search engine would never be ableto identify.

The publisher may specify a textual description for the metric as wellas a description of the units of the metric. In various implementations,each metric is a single value. In other implementations, a metric may bea set of values, such as a list of key-value pairs, which may be used toconstruct a pie chart, histogram, bar chart, etc.

The publisher may specify a wide variety of schedules based on day ofthe week, day of the month, or some other periodic or non-periodicschedule. In various implementations, the publisher can specify atriggering event that will indicate that a new value for the metric maybe available. As described in more detail below, the harvesting systemmay implement a number of optimizations, such as only acquiring a newvalue of a metric if the source of the metric has changed since themetric was last acquired.

The search engine may index and reveal information about the analyst whoproduced the metric as well as the publisher who published the metric tothe search engine. This may be used by users to assess the credibilityor scope of the metric. Further, the analyst or publisher may associatea value with a metric indicating how authoritative the metric is. Forexample, when the metric is first created, the metric may have noindicia of authority. However, over time, users of the metric or datagovernance specialists may certify that the metric can be relied upon asauthoritative. Such designation may be time-limited, lapsing after apredetermined time has passed and requiring reverification.

Special permissions may be assigned to the harvesting system so that theharvesting system can access metrics across a heterogeneousorganization. The harvesting system may associate access rights witheach metric and therefore provide the metric only to users havingsufficient access. In various implementations, the search engine may beprogrammed to inform unauthorized users of the existence of a metricwhile obscuring the value of the metric itself. In this way, users canrecognize that the metric exists and seek access if necessary for theirjob responsibilities. In traditional systems, users who do not haveaccess to certain data may assume that the metric does not exist andseek to recreate it.

Business discussions and decisions frequently rely on understandingmetrics obtained from data analysis. The explosion of data analysis evenin small organizations has quickly surpassed the ability of individualpeople to locate and catalog the analysis. In some ways, this isanalogous to the early days of the Web. However, the search enginesdeveloped for the Web are not well-positioned to obtain specificanalytics of interest to users. The increasing speed of businessdecisions demands not only the ability to identify that an analyticexists, but to obtain the metrics as quickly as possible.

A search system according to the principles of the present disclosuremay answer all of these needs, providing users with a natural languageinterface that returns metric information related to a query as well ascontext, links to source documents, author information, and anindication of freshness. Such a system may immediately expose a muchgreater set of data analysis to its users, with no limitation on itsgrowth for future data analysis. This allows the users to speed andimprove the decision-making and strategy development process by removinga number of impediments to obtaining quality metrics.

For example, in an organization that operates a high-volume pharmacy, anumber of pharmacy-benefit-related metrics and pharmacy shipping metricsmay be developed and will be of interest to a variety of users. Forexample, one metric may answer the question of how many scripts wereshipped in the prior week. Another metric may answer the question ofwhat percentage of members used a web interface for refill transactionsin the prior month. Operation of such a high-volume pharmacy, includinga number of components that produce data for analysis, is described inmore detail below.

High-Volume Pharmacy

FIG. 1 is a block diagram of an example implementation of a system 100for a high-volume pharmacy, according to an example implementation.While the system 100 is generally described as being deployed in ahigh-volume pharmacy or a fulfillment center (e.g., a mail orderpharmacy, a direct delivery pharmacy, etc., and the like), the system100 and/or components thereof may otherwise be deployed (e.g., in alower volume pharmacy, etc.). A high-volume pharmacy may be a pharmacythat is capable of filling prescriptions automatically, mechanically,manually, or a combination thereof. The system 100 may include a benefitmanager device 102, a pharmacy device 106, and one or more user devices108 in communication with each other directly and/or over a network 104.The system may also include a storage device 110.

The benefit manager device 102 is a device operated by an entity that isat least partially responsible for creation and/or management of thepharmacy or drug benefit. While such entity operating the benefitmanager device 102 is typically a pharmacy benefit manager (PBM), otherentities may operate the benefit manager device 102 on behalf ofthemselves (i.e., the PBMs) or other entities. For example, the benefitmanager device 102 may be operated by a health plan, a retail pharmacychain, a drug wholesaler, a data analytics, or other type ofsoftware-related company, etc., or the like. In some implementations, aPBM that provides the pharmacy benefit may also provide one or more thanone additional benefits including a medical or health benefit, a dentalbenefit, a vision benefit, a wellness benefit, a radiology benefit, apet care benefit, an insurance benefit, a long term care benefit, anursing home benefit, etc., and the like. The PBM may, in addition toits PBM operations, operate one or more than one pharmacies. Thepharmacies may be retail pharmacies, mail order pharmacies, etc.

Some of the operations of the PBM that operates the benefit managerdevice 102 may include the following activities and processes. A member(or a person on behalf of the member) of a pharmacy benefit plan mayobtain a prescription drug at a retail pharmacy location (e.g., alocation of a physical store, etc.) from a pharmacist or a pharmacisttechnician. The member may also obtain the prescription drug throughmail order drug delivery from a mail order pharmacy location, which maybe the high-volume pharmacy system 100. In some implementations, themember may obtain the prescription drug directly or indirectly throughthe use of a machine, such as a kiosk, vending unit, mobile electronicdevice, or a different type of mechanical, electrical, electroniccommunication device and/or computing device. Such a machine may befilled with the prescription drug in prescription packaging, which mayinclude multiple prescription components, by the high-volume pharmacysystem 100. The pharmacy benefit plan is administered by or through thebenefit manager device 102.

The member may have a copayment for the prescription drug that reflectsan amount of money that the member is responsible to pay the pharmacyfor the prescription drug. The money paid by the member to the pharmacymay come from personal funds of the member, a health savings account(HSA) of the member or the member's family, a health reimbursementarrangement (HRA) of the member or the member's family, a flexiblespending account (FSA) of the member or the member's family, etc., orthe like. In some instances, an employer of the member may directly orindirectly fund or reimburse the member for the copayments.

The amount of the copayment required by the member may vary withdifferent pharmacy benefit plans having different plan sponsors orclients and/or for different prescription drugs. The member's copaymentmay be a flat copayment (e.g., $10, etc.), co-insurance (e.g., 10%,etc.), and/or a deductible (e.g., for first $500 of annual prescriptiondrug expense, etc.) for certain prescription drugs, certain types and/orclasses of prescription drugs, and/or all prescription drugs. Thecopayment may be stored in the storage device 110 or determined by thebenefit manager device 102.

In some instances, the member may not pay the copayment or may only paya portion of the copayment for the prescription drug. For example, if ausual and customary cost for a generic version of a prescription drug is$4, and the member's flat copayment is $20 for the prescription drug,the member may only need to pay $4 to receive the prescription drug. Inanother example involving a worker's compensation claim, no copaymentmay be due by the member for the prescription drug.

In addition, copayments may also vary based on different deliverychannels used for the prescription drug to be received. For example, thecopayment for receiving the prescription drug from a mail order pharmacylocation may be less than the copayment for receiving the prescriptiondrug from a retail pharmacy location.

In conjunction with receiving a copayment (if any) from the member anddispensing the prescription drug to the member, the pharmacy submits aclaim to the PBM for the prescription drug. After receiving the claim,the PBM, e.g., the benefit manager device 102, may perform certainadjudication operations including verifying eligibility for the member,identifying/reviewing an applicable formulary for the member todetermine any appropriate copayment, coinsurance, and deductible for theprescription drug, and performing a drug utilization review (DUR) on themember. Further, the PBM may provide a response to the pharmacy, e.g.,the pharmacy system 100, following performance of at least some of theaforementioned operations.

As part of the adjudication, a plan sponsor (or the PBM on behalf of theplan sponsor) ultimately reimburses the pharmacy for filling theprescription drug when the prescription drug was successfullyadjudicated.

The aforementioned adjudication operations generally occur before thecopayment is received and the prescription drug is dispensed. However insome instances, these operations may occur simultaneously, substantiallysimultaneously, or in a different order. In addition, more or lessadjudication operations may be performed as at least part of theadjudication process.

The amount of reimbursement paid to the pharmacy by a plan sponsorand/or money paid by the member may be determined at least partiallybased on types of pharmacy network in which the pharmacy is included. Insome implementations, the amount may also be determined based on otherfactors. For example, if the member pays the pharmacy for theprescription drug without using the prescription or drug benefitprovided by the PBM, the amount of money paid by the member may behigher than when the member uses the prescription or drug benefit. Insome implementations, the amount of money received by the pharmacy fordispensing the prescription drug and for the prescription drug itselfmay be higher than when the member uses the prescription or drugbenefit. Some or all of the foregoing operations may be performed byexecuting instructions stored in the benefit manager device 102 and/oran additional device.

Examples of the network 104 include Mobile Communications (GSM) network,a code division multiple access (CDMA) network, 3rd GenerationPartnership Project (3GPP), an Internet Protocol (IP) network, aWireless Application Protocol (WAP) network, a WiFi network, or an IEEE802.11 standards network, as well as various combinations thereof. Thenetwork 104 may include an optical network. The network 104 may be alocal area network or a global communication network, such as theInternet. In some implementations, the network 104 may include a networkdedicated to prescription orders, e.g., a prescribing network such asthe electronic prescribing network operated by Surescripts of Arlington,Va.

Moreover, although the system shows a single network 104, multiplenetworks can be used. The multiple networks may communicate in serieswith each other to link the devices 102-110 or in parallel to link thedevices 102-110.

The pharmacy device 106 may be a device associated with a retailpharmacy location (e.g., an exclusive pharmacy location, a grocery storewith a retail pharmacy, or a general sales store with a retail pharmacy)or other type of pharmacy location at which a member attempts to obtaina prescription. The pharmacy device 106 may be utilized by the pharmacyto submit the claim to the PBM for adjudication.

Additionally, in some implementations, the pharmacy device 106 mayenable information exchange between the pharmacy and the PBM, forexample, to allow the sharing of member information such as drughistory, and the like, that may allow the pharmacy to better service amember (e.g., by providing more informed therapy consultation and druginteraction information, etc.). In some implementations, the benefitmanager device 102 may track prescription drug fulfillment and/or otherinformation for users that are not members or have not identifiedthemselves as members, at the time (or in conjunction with the time) inwhich they seek to have a prescription filled at a pharmacy.

The pharmacy device 106 may include a pharmacy fulfillment device 112,an order processing device 114, and a pharmacy management device 116 incommunication with each other directly and/or over the network 104.

The order processing device 114 may receive information regardingfilling prescriptions and may direct an order component to one or moredevices of the pharmacy fulfillment device 112 at a pharmacy. Thepharmacy fulfillment device 112 may fulfill, dispense, aggregate, and/orpack the order components of the prescription drugs in accordance withone or more prescription orders directed by the order processing device114. The order processing device 114 may be deployed in the system 100,or may otherwise be used.

In general, the order processing device 114 is a device located withinor otherwise associated with the pharmacy to enable fulfillment of aprescription and dispensing prescription drugs by the pharmacyfulfillment device 112. In some implementations, the order processingdevice 114 may be an external order processing device separate from thepharmacy and communicate with other devices located within the pharmacy.

For example, the external order processing device may communicate withan internal pharmacy order processing device and/or other deviceslocated within the system 100. In some implementations, the externalorder processing device may have limited functionality (e.g., asoperated by a user requesting fulfillment of a prescription drug), whilethe internal pharmacy order processing device may have greaterfunctionality (e.g., as operated by a pharmacist).

The order processing device 114 may track the prescription order as itis fulfilled by the pharmacy fulfillment device 112. The prescriptionorder may include one or more than one prescription drugs to be filledby the pharmacy. The order processing device 114 may make pharmacyrouting decisions and/or order consolidation decisions for theparticular prescription order. The pharmacy routing decisions includewhat device(s) in the pharmacy are responsible for filling or otherwisehandling certain portions of the prescription order. The orderconsolidation decisions include whether portions of one prescriptionorder or multiple prescription orders should be shipped together for auser or a user family. The order processing device 114 may also trackand/or schedule literature or paperwork associated with eachprescription order or multiple prescription orders that are beingshipped together. In some implementations, the order processing device114 may operate in combination with the pharmacy management device 116.

The order processing device 114 may include circuitry, a processor, amemory to store data and instructions, and communication functionality.The order processing device 114 is dedicated to performing processes,methods and/or instructions described herein. Other types of electronicdevices specifically configured to implement with the processes, methodsand/or instructions described herein may also be used.

In some implementations, at least some functionalities of the orderprocessing device 114 may be included in the pharmacy management device116. The order processing device 114 may be in a client-serverrelationship with the pharmacy management device 116, in a peer-to-peerrelationship with the pharmacy management device 116, or in a differenttype of relationship with the pharmacy management device 116. The orderprocessing device 114 and/or the pharmacy management device 116 maycommunicate directly (e.g., by utilizing a local storage, etc.) and/orthrough the network 104 (e.g., by utilizing a cloud configuration orsoftware as a service. etc.) with the storage device 110.

The user device 108 is used by a device operator. The device operatormay be a user (e.g., an employee, a contractor, a benefit member, etc.)associated with a software development project. Other device operatorsmay also operate the user device 108. The user device 108 may be astand-alone device that solely provides at least some of thefunctionality to enable analysis of software development risks, or maybe a multi-use device that has functionality outside of analysis ofsoftware development risks. Examples of the user device 108 include aset-top box (STB), a receiver card, a mobile telephone, a personaldigital assistant (PDA), a display device, a portable gaming unit, and acomputing system, etc. Other devices, however, may also be used. In someembodiments, the computing system may include a mobile computing device.For example, the user device 108 may include a mobile electronic device,such an iPhone or iPad by Apple, Inc., mobile electronic devices poweredby Android by Google, Inc., and a Blackberry by Research In MotionLimited. The user device 108 may also include other computing devices,such as desktop computing devices, notebook computing devices, netbookcomputing devices, gaming devices, and the like. Other types ofelectronic devices may also be used.

The storage device 110 may include: a non-transitory storage (e.g.,memory, hard disk, CD-ROM, etc.) in communication with the benefitmanager device 102, the pharmacy device 106, and/or the user device 108directly and/or over the network 104. The non-transitory storage maystore order data 118, member data 120, claims data 122, drug data 124,prescription data 126, and/or plan sponsor data 128. Further, the system100 may include additional devices, which may communicate with eachother directly or over the network 104.

The order data 118 may be related to a prescription order. The orderdata may include type of the prescription drug (e.g., drug name andstrength, etc.) and quantity of the prescription drug, etc. The orderdata 118 may also include data used for completion of the prescription,such as prescription materials. In general, prescription materialsinclude an electronic copy of information regarding the prescriptiondrug for inclusion with or otherwise in conjunction with the fulfilledprescription. The prescription materials may include electronicinformation regarding drug interaction warnings, recommended usage,possible side effects, expiration date, date of prescribing, or thelike. The order data 118 may be used by a high volume fulfillment centerto fulfill a pharmacy order.

In some implementations, the order data 118 includes verificationinformation associated with fulfillment of the prescription in thepharmacy. For example, the order data 118 may include videos and/orimages taken of (i) the prescription drug prior to dispensing, duringdispensing, and/or after dispensing, (ii) the prescription container(e.g., a prescription container and sealing lid, prescription packagingand the like) used to contain the prescription drug prior to dispensing,during dispensing, and/or after dispensing, (iii) the packaging and/orpackaging materials used to ship or otherwise deliver the prescriptiondrug prior to dispensing, during dispensing, and/or after dispensing,and/or (iv) the fulfillment process within the pharmacy. Other types ofverification information such as bar code data read from pallets, bins,trays, carts, and the like used to transport prescriptions within thepharmacy may also be stored as order data 118.

The member data 120 includes information regarding the membersassociated with the PBM. The information stored as member data 120 mayinclude personal information, personal health information, protectedhealth information, and the like. Examples of the member data 120include name, address, telephone number, e-mail address, prescriptiondrug history, etc., and the like. The member data 120 may include a plansponsor identifier that identifies the plan sponsor associated with themember and/or a member identifier that identifies the member to the plansponsor. The member data 120 may include a member identifier thatidentifies the plan sponsor associated with the user and/or a useridentifier that identifies the user to the plan sponsor. The member data120 may also include, by way of example, dispensation preferences suchas type of label, type of cap, message preferences, languagepreferences, or the like.

The member data 120 may be accessed by various devices in the pharmacy,(e.g., the high volume fulfillment center, etc.), to obtain informationutilized for fulfillment and shipping of prescription orders. In someimplementations, an external order processing device operated by or onbehalf of a member may have access to at least a portion of the memberdata 120 for review, verification, etc., or other purposes.

In some implementations, the member data 120 may include information forpersons who are users of the pharmacy but are not members in thepharmacy benefit plan being provided by the PBM. For example, theseusers may obtain drug directly from the pharmacy, through a privatelabel service offered by the pharmacy, the high volume fulfillmentcenter, or otherwise. In general, the use of the terms member and usermay be used interchangeably herein.

The claims data 122 includes information regarding pharmacy claimsadjudicated by the PBM under a drug benefit program provided by the PBMfor one, or more than one, plan sponsors. In general, the claims data122 includes an identification of the client that sponsors the drugbenefit program under which the claim is made, and/or the member thatpurchased the prescription drug giving rise to the claim, theprescription drug that was filled by the pharmacy (e.g., the nationaldrug code number, etc.), the dispensing date, generic indicator, GPInumber, medication class, the cost of the prescription drug providedunder the drug benefit program, the copay/coinsurance amount, rebateinformation, and/or member eligibility, etc. Additional information maybe included.

In some implementations, other types of claims beyond prescription drugclaims may be stored in the claims data 122. For example, medicalclaims, dental claims, wellness claims, or other type of healthcare-related claims for members may be stored as a portion of the claimsdata 122.

In some implementations, the claims data 122 includes claims thatidentify the members with whom the claims are associated. In someimplementations, the claims data 122 includes claims that have beende-identified (e.g., associated with a unique identifier but not with aparticular, identifiable member, etc.).

The drug data 124 may include drug name (e.g., technical name and/orcommon name, etc.), other names by which the drug is known by, activeingredients, an image of the drug (e.g., in pill form, etc.), and thelike. The drug data 124 may include information associated with a singlemedication or multiple medications.

The prescription data 126 may include information regardingprescriptions that may be issued by prescribers on behalf of users, whomay be members of the pharmacy benefit plan, for example to be filled bya pharmacy. Examples of the prescription data 126 include user names,medication or treatment (such as lab tests), dosing information, and thelike. The prescriptions may be electronic prescriptions, paperprescriptions that have been scanned, or otherwise. In someimplementations, the dosing information reflects a frequency of use(e.g., once a day, twice a day, before each meal, etc.) and a durationof use (e.g., a few days, a week, a few weeks, a month, etc.).

In some implementations, the order data 118 may be linked to associatedmember data 120, claims data 122, drug data 124, and/or prescriptiondata 126.

The plan sponsor data 128 includes information regarding the plansponsors of the PBM. Examples of the plan sponsor data 128 includecompany name, company address, contact name, contact telephone number,contact e-mail address, etc., and the like.

FIG. 2 illustrates the pharmacy fulfillment device 112, according to anexample implementation. The pharmacy fulfillment device 112 may be usedto process and fulfill prescriptions and prescription orders. Afterfulfillment, the fulfilled prescriptions are packed for shipping.

The pharmacy fulfillment device 112 may include devices in communicationwith the benefit manager device 102, the order processing device 114,and/or the storage device 110, directly or over the network 104.Specifically, the pharmacy fulfillment device 112 may include palletsizing and pucking device(s) 206, loading device(s) 208, inspectdevice(s) 210, unit of use device(s) 212, automated dispensing device(s)214, manual fulfillment device(s) 216, review devices 218, imagingdevice(s) 220, cap device(s) 222, accumulation devices 224, packingdevice(s) 226, literature device(s) 228, unit of use packing device(s)230, and mail manifest device(s) 232. Further, the pharmacy fulfillmentdevice 112 may include additional devices, which may communicate witheach other directly or over the network 104.

In some implementations, operations performed by one of these devices206-232 may be performed sequentially, or in parallel with theoperations of another device as may be coordinated by the orderprocessing device 114. In some implementations, the order processingdevice 114 tracks a prescription with the pharmacy based on operationsperformed by one or more than one devices 206-232.

In some implementations, the pharmacy fulfillment device 112 maytransport prescription drug containers, for example, between more thanone of the devices 206-232 in the high volume fulfillment center, by useof pallets. The pallet sizing and pucking device 206 may configure pucksin a pallet. A pallet may be a transport structure for a number ofprescription containers, and may include a number of cavities. A puckmay be placed in one or more than one of the cavities in a pallet by thepallet sizing and pucking device 206. The puck may include a receptaclesized and shaped to receive a prescription container. Such containersmay be supported by the pucks during carriage in the pallet. Differentpucks may have differently sized and shaped receptacles to accommodatecontainers of differing sizes, as may be appropriate for differentprescriptions.

The arrangement of pucks in a pallet may be determined by the orderprocessing device 114 based on prescriptions that the order processingdevice 114 decides to launch. The arrangement logic may be implementeddirectly in the pallet sizing and pucking device 206. Once aprescription is set to be launched, a puck suitable for the appropriatesize of container for that prescription may be positioned in a pallet bya robotic arm or pickers. The pallet sizing and pucking device 206 maylaunch a pallet once pucks have been configured in the pallet.

The loading device 208 may load prescription containers into the puckson a pallet by a robotic arm, a pick and place mechanism, or the like.In various implementations, the loading device 208 has robotic arms orpickers to grasp a prescription container and move it to and from apallet or a puck. The loading device 208 may also print a label that isappropriate for a container that is to be loaded onto the pallet, andapply the label to the container. The pallet may be located on aconveyor assembly during these operations, (e.g., at the high volumefulfillment center, etc.).

The inspect device 210 may verify that containers in a pallet arecorrectly labeled and in the correct spot on the pallet. The inspectdevice 210 may scan the label on one or more than one containers on thepallet. Labels of containers may be scanned or imaged in full or in partby the inspect device 210. Such imaging may occur after the containerhas been lifted out of its puck by a robotic arm, picker, etc., or thelike, or may be otherwise scanned or imaged while retained in the puck.In some implementations, images and/or video captured by the inspectdevice 210 may be stored in the storage device 110 as order data 118.

The unit of use device 212 may temporarily store, monitor, label and/ordispense unit of use products. In general, unit of use products areprescription drug products that may be delivered to a user or memberwithout being repackaged at the pharmacy. These products may includepills in a container, pills in a blister pack, inhalers, and the like.Prescription drug products dispensed by the unit of use device 212 maybe packaged individually or collectively for shipping, or may be shippedin combination with other prescription drugs dispensed by other devicesin the high volume fulfillment center.

At least some of the operations of devices 206-232 may be directed bythe order processing device 114. For example, the manual fulfillmentdevice 216, the review device 218, the automated dispensing device 214,and/or the packing device 226, etc. may receive instructions provided bythe order processing device 114.

The automated dispensing device 214 may include one or more than onedevices that dispense prescription drugs or pharmaceuticals intoprescription containers in accordance with one or multiple prescriptionorders. In general, the automated dispensing device 214 may includemechanical and electronic components with, in some implementations,software and/or logic to facilitate pharmaceutical dispensing that wouldotherwise be performed in a manual fashion by a pharmacist and/orpharmacist technician. For example, the automated dispensing device 214may include high volume fillers that fill a number of prescription drugtypes at a rapid rate and blister pack machines that dispense and packdrugs into a blister pack. Prescription drugs dispensed by the automateddispensing devices 214 may be packaged individually or collectively forshipping, or may be shipped in combination with other prescription drugsdispensed by other devices in the high volume fulfillment center.

The manual fulfillment device 216 may provide for manual fulfillment ofprescriptions. For example, the manual fulfillment device 216 mayreceive or obtain a container and enable fulfillment of the container bya pharmacist or pharmacy technician. In some implementations, the manualfulfillment device 216 provides the filled container to another devicein the pharmacy fulfillment devices 112 to be joined with othercontainers in a prescription order for a user or member. In general, amanual fulfillment may include operations at least partially performedby a pharmacist or a pharmacy technician. For example, a person mayretrieve a supply of the prescribed drug, may make an observation, maycount out a prescribed quantity of drugs and place them into aprescription container, or the like. Some portions of the manualfulfillment process may be automated by use of a machine. For example,counting of capsules, tablets, or pills may be at least partiallyautomated (e.g., through use of a pill counter, etc.). Prescriptiondrugs dispensed by the manual fulfillment device 216 may be packagedindividually or collectively for shipping, or may be shipped incombination with other prescription drugs dispensed by other devices inthe high volume fulfillment center.

The review device 218 may process prescription containers to be reviewedby a pharmacist for proper pill count, exception handling, prescriptionverification, and the like. Fulfilled prescriptions may be manuallyreviewed and/or verified by a pharmacist, as may be required by state orlocal law. A pharmacist or other licensed pharmacy person who maydispense certain drugs in compliance with local and/or other laws mayoperate the review device 218 and visually inspect a prescriptioncontainer that has been filled with a prescription drug. The pharmacistmay review, verify, and/or evaluate drug quantity, drug strength, and/ordrug interaction concerns, or otherwise perform pharmacist services. Thepharmacist may also handle containers which have been flagged as anexception, such as containers with unreadable labels, containers forwhich the associated prescription order has been canceled, containerswith defects, and the like. In an example, the manual review can beperformed at the manual station.

The imaging device 220 may image containers once they have been filledwith pharmaceuticals. The imaging device 220 may measure a fill heightof the pharmaceuticals in the container based on the obtained image todetermine if the container is filled to the correct height given thetype of pharmaceutical and the number of pills in the prescription.Images of the pills in the container may also be obtained to detect thesize of the pills themselves and markings thereon. The images may betransmitted to the order processing device 114, and/or stored in thestorage device 110 as part of the order data 118.

The cap device 222 may be used to cap or otherwise seal a prescriptioncontainer. In some implementations, the cap device 222 may secure aprescription container with a type of cap in accordance with a userpreference (e.g., a preference regarding child resistance, etc.), a plansponsor preference, a prescriber preference, or the like. The cap device222 may also etch a message into the cap, although this process may beperformed by a subsequent device in the high volume fulfillment center.

The accumulation device 224 accumulates various containers ofprescription drugs in a prescription order. The accumulation device 224may accumulate prescription containers from various devices or areas ofthe pharmacy. For example, the accumulation device 224 may accumulateprescription containers from the unit of use device 212, the automateddispensing device 214, the manual fulfillment device 216, and the reviewdevice 218, at the high volume fulfillment center. The accumulationdevice 224 may be used to group the prescription containers prior toshipment to the member or otherwise.

The literature device 228 prints, or otherwise generates, literature toinclude with prescription drug orders. The literature may be printed onmultiple sheets of substrates, such as paper, coated paper, printablepolymers, or combinations thereof. The literature printed by theliterature device 228 may include information required to accompany theprescription drugs included in a prescription order, relating toprescription drugs in the order, financial information associated withthe order (e.g., an invoice or an account statement, etc., or the like).

In some implementations, the literature device 228 folds or otherwiseprepares the literature for inclusion with a prescription drug order(e.g., in a shipping container, etc.). In some implementations, theliterature device 228 that prints the literature may be separate fromthe literature device that prepares the literature for inclusion with aprescription order.

The packing device 226 packages a prescription order in preparation forshipping the order. The packing device 226 may box, bag, or otherwisepackage the fulfilled prescription order for delivery. The packingdevice 226 may further place inserts, (e.g., literature or other papers,etc.), into the packaging received from the literature device 228 orotherwise. For example, bulk prescription orders may be shipped in abox, while other prescription orders may be shipped in a bag which maybe a wrap seal bag.

The packing device 226 may label the box or bag with an address and arecipient's name. The label may be printed and affixed to the bag orbox, be printed directly onto the bag or box, or otherwise associatedwith the bag or box, etc. The packing device 226 may sort the box or bagfor mailing in an efficient manner (e.g., sort by delivery address,etc.). The packing device 226 may include ice or temperature sensitiveelements for prescriptions which are to be kept within a temperaturerange during shipping in order to retain efficacy or otherwise. Theultimate package may then be shipped through postal mail, through a mailorder delivery service that ships via ground and/or air (e.g., UPS,FEDEX, or DHL, etc.), through delivery service, through a locker box ata shipping site (e.g., AMAZON locker or a PO Box, etc.), or otherwise.

The unit of use packing device 230 packages a unit of use prescriptionorder in preparation for shipping the order. The unit of use packingdevice 230 may include manual scanning of containers to be bagged forshipping to verify each container in the order. In an exampleimplementation, the manual scanning may be performed at a manualstation. The pharmacy fulfillment device 112 may also include a mailmanifest device 232 to print mailing labels used by the packing device226 and may print shipping manifests and packing lists.

While the pharmacy fulfillment device 112 in FIG. 2 is shown to includesingle devices 206-232, multiple devices may be used. The devices206-232 may be the same type or model of device or may be differentdevice types or models. When multiple devices are present, the multipledevices may be of the same device type or models or may be a differentdevice type or model. The types of devices 206-232 shown in FIG. 2 areexample devices. In other configurations of the system 100, lesser,additional, or different types of devices may be included.

Moreover, multiple devices may share processing and/or memory resources.The devices 206-232 may be located in the same area or in differentlocations. For example, the devices 206-232 may be located in a buildingor set of adjoining buildings. The devices 206-232 may be interconnected(e.g. by conveyors, etc.), networked, and/or otherwise in contact withone another or integrated with one another, (e.g., at the high volumefulfillment center, etc.). In addition, the functionality of a devicemay be split among a number of discrete devices and/or combined withother devices.

FIG. 3 illustrates the order processing device 114, according to anexample implementation. The order processing device 114 may be used byone or more than one operators to generate prescription orders, makerouting decisions, make prescription order consolidation decisions,track literature with the system 100, and/or view order status and otherorder related information. For example, the prescription order may becomprised of order components.

The order processing device 114 may receive instructions to fulfill anorder without operator intervention. An order component may include aprescription drug fulfilled by use of a container through the system100. The order processing device 114 may include an order verificationsubsystem 302, an order control subsystem 304, and/or an order trackingsubsystem 306. Other subsystems may also be included in the orderprocessing device 114.

The order verification subsystem 302 may communicate with the benefitmanager device 102 to verify the eligibility of the member and reviewthe formulary to determine appropriate copayment, coinsurance, anddeductible for the prescription drug and/or perform a DUR. Othercommunications between the order verification subsystem 302 and thebenefit manager device 102 may be performed for a variety of purposes.

The order control subsystem 304 controls various movements of thecontainers and/or pallets along with various filling functions duringtheir progression through the system 100. In some implementations, theorder control subsystem 304 may identify the prescribed drug in one ormore than one prescription orders as capable of being fulfilled by theautomated dispensing device 214. The order control subsystem 304 maydetermine which prescriptions are to be launched and may determine thata pallet of automated-fill containers is to be launched.

The order control subsystem 304 may determine that an automated-fillprescription of a specific pharmaceutical is to be launched and mayexamine a queue of orders awaiting fulfillment for other prescriptionorders which will be filled with the same pharmaceutical. The ordercontrol subsystem 304 may then launch orders with similar automated-fillpharmaceutical needs together in a pallet to the automated dispensingdevice 214. As the devices 206-232 may be interconnected by a system ofconveyors or other container movement systems, the order controlsubsystem 304 may control various conveyors to deliver the pallet fromthe loading device 208 to the manual fulfillment device 216, forexample, from the literature device 228 to deliver paperwork as neededto fill the prescription.

The order tracking subsystem 306 may track a prescription order as itprogresses (or stops) toward fulfillment. The order tracking subsystem306 may track, record and/or update order history, order status, or thelike. The order tracking subsystem 306 may store data locally (e.g., ina memory, etc.) or as a portion of the order data 118 stored in thestorage device 110.

Block Diagrams

In FIG. 4, an example environment in which the principles of the presentdisclosure can be implemented includes analysis engines 404-1 and 404-2(collectively, analysis engines 404). Although two analysis engine areshown in FIG. 4, the present disclosure is equally applicable to asingle analysis engine as well as to more than two analysis engines.Each of the analysis engines 404 may include a client computing deviceor a server computing device that may execute database software,business intelligence (BI) software, and/or custom analysis software.

In various implementations, the analysis engines 404 may be operated forthe benefit of a single organization, such as a corporation. Theanalysis engines 404 may be operated wholly or partially on equipmentowned by the organization, which may be located on premises owned by theorganization or on premises provided by a third party. In variousimplementations, one or both of the analysis engines 404 may be operatedand hosted by a third party. For example, the analysis engine 404-1 maybe a software-as-a-service offering from a third party that operates onthe third party's servers and uses the organization's data.

An analyst device 408, which may include a laptop or desktop computerunder the control of an analyst, controls the analysis engine 404-1. Forexample, the analyst device 408 may establish a database schema,configure parameters for extract, transform, and load (ETL) procedures,etc. The analyst device 408 may configure data processing by theanalysis engine 404-1 that results in the generation of one or moremetrics.

These metrics may be outputted, such as to files on a file server 412.For example, the metrics may be outputted as a text-based file, such asa comma separated value (CSV) file. In other configurations, the fileserver 412 may host proprietary file formats such as spreadsheet ordatabase files. These files may be outputted by the analysis engine404-1 or may store data for use by the analysis engine 404-1. In someconfigurations, certain files on the file server 412 are reserved foroutput, such as of metrics. In other configurations, files on the fileserver 412 may combine metrics with input and intermediate data for useby the analysis engine 404-1.

Although solid lines are shown in FIG. 4 to indicate logical data flowamong elements, actual communication between these elements may beindirect. For example, the elements may communicate with each other viaa communications system 420. Because the exact physical interconnectionis not central to the disclosure, the physical interconnections areshown in dashed lines and omitted in future figures.

The analysis engine 404-2 interfaces with a data store 424, which may bethe backend for a relational database or a NoSQL database. A publisherdevice 428 is a computing device under the control of a publisher who isresponsible for making the results of analysis available to other users.In various implementations, the publisher may be the same as the analystwho created the data or their roles may be separate. An analyst may havedeveloped a structured query language (SQL) query to run on the analysisengine 404-2 that results in a metric.

Via the publisher device 428, the publisher may publish the SQL query toa search system according to the principles of the present disclosure.As an example implementation only, the search system includes a webserver 432 used for administration and onboarding of metrics, aninstruction data store 436, an access mechanism store 440, a harvestingsystem 444, a search engine 448, and a web server 452 accessed by users.In various implementations, the web server 432 and the web server 452may be implemented by a common web server architecture, such as an Nginxweb server from Nginx, Inc., and may simply be two different pagesserved from the common web server architecture.

The web server 432 presents an administrative interface that allows thepublisher, via the publisher device 428, to specify the SQL query andindicate how to identify a particular metric from the result of the SQLquery. In addition, the publisher may provide metadata to the web server432 about the metric. For example, this metadata may include a schedulefor when the metric is updated. Updates may occur on a strictly periodicbasis or according to a calendar, or may only occur in response tocertain events.

The publisher may provide a textual description of the metric in fullsentence form as well as a short form text description. The publishermay also provide a set of tags by which the metric can be indexed. Thepublisher device 428 may indicate an identity of the analyst or analysisteam that generated the SQL query. The publisher may provide contactinformation for the publisher as well as the analyst to allow consumersof the metric to request additional information regarding the metric orto request similar metrics. The publisher may also assign a confidencescore to the metric indicating the degree to which the metric isauthoritative. The confidence score may also indicate the degree towhich the publisher is confident that the metric will remain valid overtime.

The publisher may specify access rights for the metric. As one example,the access rights may be co-extensive with the access rights to theanalysis engine 404-2 or the data store 424. In other scenarios, thepublisher may provide wider access to the metric than is available forthe underlying data. In various implementations, the search system mayidentify what access rights apply to a metric and then automaticallyrestrict disclosure of that metric to users having the same accessrights.

In various implementations, some or all of the information provided bythe publisher/analyst may be stored within the metric source itselfrather than being provided to the instruction data store 436. Forexample, the publisher may simply provide the web server 432 with alocation of a metric and an update schedule. According to the updateschedule, the harvesting system 444 can access the source of the metricat the specified location and obtain the remaining data, such as textdescription, keywords, access rights, etc. In this way, an analyst canupdate the metric metadata without needing to use the web server 432 oreven having access rights to the web server 432.

Meanwhile, via the analyst device 408, the analyst may provide similarmetric information to the web server 432 with respect to a metricavailable from the analysis engine 404-1. Although the illustration inFIG. 4 depicts a single analyst device (408) communicating with a singleanalysis engine (404-1) and a single publisher device (428)communicating with another analysis engine (404-2), there is nothing torestrict analyst or publisher devices from communicating with multipleanalysis engines. In other words, a publisher, using a publisher device,may publish multiple metrics from across a variety of analysis engines.Similarly, an analyst, using an analyst device, may perform analysisusing a variety of analysis engines as well as publish metrics from avariety of analysis engines.

The web server 432 stores the metric information provided by thepublisher into the instruction data store 436. For example only, theinstruction data store 436 may be implemented using a PostgreSQLdatabase system from the PostgreSQL Global Development Group. Theharvesting system 444, as described in more detail below, usesinstructions from the instruction data store 436—such as a uniformresource locator (URL) or a SQL query—to obtain values of the metricsaccording to the schedule specified in the instruction data store 436.

The harvesting system 444 is configured with access mechanisms defininghow to access metrics from across a wide array of source data. Theseaccess mechanisms may be stored in the access mechanism store 440. Forexample, the access mechanism store 440 may include parsing routinesconfigured to extract data from various files, such as standardized orproprietary spreadsheet formats, text-based formats (such as CSV orXML), word processing documents, publishing documents (such as PDFs),etc.

The access mechanism store 440 may also include access mechanisms, suchas a library or executable, configured to obtain information frombusiness intelligence (BI) software using an application programminginterface (API). In other implementations, the access mechanism store440 may access an API that causes the BI software to export certaindata, such as to a text file on a file server. The access mechanismstore 440 may also store access mechanisms configured to submit andreceive responses to SQL queries. The access mechanism store 440 mayalso include libraries for accessing online resources, such as web pagesor SHAREPOINT collaboration site documents.

Further, the access mechanism store 440 may include authenticationinformation, such as authentication tokens, that authorize theharvesting system 444 to access various systems such as file servers anddatabases. In various implementations, the access mechanism store 440may be maintained by an administrator of the search system. Additionallyor alternatively, analysts and publishers may use the web server 432 toprovide access mechanism information to the access mechanism store 440.For example, a publisher may create access credentials to a database andprovide those credentials to the access mechanism store 440 for ongoingaccess to the database.

As the harvesting system 444 acquires metric information according tothe instruction data store 436, the harvesting system 444 storesacquired metrics into a metric data store 456 of the search engine 448.In various implementations, the search engine 448 may index datareceived from the harvesting system 444 and store only these indices inthe metric data store 456. In other words, the indices themselves storedata and are not simply an index of a separate repository of metrics.The metric data store 456 may encrypt some or all metricvalues—particularly, personally identifiable information, patientinformation, and financial data.

The search engine 448 provides information from the metric data store456 upon request. For example, users may find a web interface mostconvenient and therefore a web server 452 provides a front-end to thesearch engine 448. The web server 452 receives queries, such as fromuser devices 460-1 and 460-2 (collectively, user devices 460). The webserver 452 provides information to the search engine 448 regarding thequeries and presents results to the user devices 460.

The web server 452 may enforce access controls for the user devices 460,and may therefore require authentication by the user devices 460. Invarious implementations, certain data may be publicly accessible to anydevices that are able to communicate with the web server 452. Therefore,the user devices 460 may only need to authenticate to the web server 452to obtain data having more restricted access rights.

The web server 452 may display the most recent metric value for each ofthe search results. In various implementations, each of the user devices460 may be able to configure the web server 452 to automatically provideadditional historical metrics immediately or only after requestinghistorical data. As described above, the web server 452 may provide textdescriptions of all relevant search results regardless of access rightsbut restrict display of the metric values to those having sufficientaccess rights. In addition, access rights may be more granular thansimply by metric. For example, access rights may specify that a user cansee the most recent value for a metric but not all of the historicalvalues.

The web server 452 may also display, either immediately or upon arequest for additional data, metadata associated with metrics, such asupdate time and source contact information. In addition, the web server452 may allow the user devices 460 to quickly obtain access to theunderlying source location or document. As an example, this may take theform of a hyperlink to a web server, file system, etc.

The web server 452 may receive information from the user devices 460,such as whether associated users of the user devices 460 believe acertain metric to be authoritative and/or helpful. This feedback may bedisplayed for other users to make value judgments about a metric. Theweb server 452 may also be configured to visually identify metrics thathave been confirmed as authoritative, such as with a certain icon orchanges in font formatting, such as application of a bold font face. Invarious implementations, the web server 452 may be configuredauthoritative metrics more highly.

FIG. 5 presents elements of an example implementation of the searchengine 448. A natural language processing module 504 receives a query,such as from the web server 452 of FIG. 4. Natural language processingmodule 504 determines a user intent and provides text tokens related tothe user intent to a search module 508. As an example, the search module508 may be implemented using the APACHE LUCENE information retrievallibrary from the Apache Software Foundation. The natural languageprocessing module 504 may be implemented using the query parser of theApache Lucene information retrieval library, which breaks a query intosingle terms, phrases, and operators.

The search module 508 identifies metrics from the metric data store 456according to the tokens from the natural language processing module 504.For example, the tokens may be compared to keywords and text associatedwith the metrics in the metric data store 456. In some implementations,these keywords and text may be obtained from the instruction data store436. In other implementations, these keywords and text are obtained fromthe source of the metric and stored into the metric data store 456.

A single set of metadata may apply to all values of a metric over time;or, taking into account that the metadata may change over time, metadatamay be stored with each value of the metric. As an implementationdetail, the metric data store 456 may use a compression scheme (forexample, similar to run length encoding) so that multiple copies ofidentical metadata do not need to be saved with each metric acquired.

A results presentation module 512 generates hypertext markup language(HTML) results based on metrics obtained by the search module 508. Theseresults include information about the metric as well as the metricitself. In addition, the results presentation module 512 may visuallyidentify metrics determined to be authoritative by a data governancedecoration module 516. The data governance decoration module 516 mayrely on information from the metric data store 456 specified bypublishers and/or data governance specialists.

A data expansion module 520 may select a single value of the metric forthe results presentation module 512—generally, the most recent value ofthe metric—with the ability to expand to additional values of themetric. For example, the data expansion module 520 may construct a linechart or bar chart of historical values of the metric in response to arequest from a user for additional data.

The results presentation module 512 may initially provide a textualdescription of the metric along with the value for the metric. Upon userselection of metric, additional data may be displayed, such as areference to the source data and contact information for the analyst orpublisher, which may allow a direct email, call, or messaging platformcommunication to be initiated by the user.

In FIG. 6, an example implementation of the harvesting system 444includes a scheduling system 604 that determines, for each metric, whento check for a new value of the metric. The scheduling system 604 relieson information from the instruction data store 436 and may use schedulesdefined in a schedule store 608. For example, the schedule store 608 mayencode schedules such as the first Monday of the month, the last Fridayof the month, the first and 15th of each month, etc.

Upon request by the scheduling system 604, an acquisition system 612attempts to obtain the metric specified by the instruction data store436. For example, the acquisition system 612 may communicate with a fileserver, a database, an online resource (such as a SharePointcollaboration server from Microsoft Corporation) to obtain the metric.

In various implementations, the acquisition system 612 may first checkto determine whether a modification date is more recent than the time atwhich the prior value of the metric was obtained. For example, theaccess mechanism store 440 may specify whether and how to determine amodification date. For example, an operating system may maintain amodification date for each file in the file system. If that modificationdate is not newer than the prior acquisition time of the metric, theacquisition system 612 may skip obtaining the metric.

In other implementations, the data source may encode the modificationdate in a response, such as to a SQL query or API call. However, ifobtaining the modification date will require as much processing powerand create the same amount of latency as simply obtaining the metricdirectly, checking the modification date may be avoided.

In various implementations, the publisher may specify that the metric beobtained regardless of whether the date or data has changed. Forexample, the operating system modification date may be inaccurate orunreliable. In another example, the source of a metric may update anoutput file only when the value of the metric changes. Therefore, eventhough the metric value has not changed and the modification date hasnot been updated, this represents a new value of the metric that shouldbe indexed and presented to users.

Once data (such as the results of a SQL query) or a file is obtained bythe acquisition system 612, a data processing module 620 may parse thedata or file to extract the metric. This metric value is then providedto the metric data store 456. The data processing module 620 may operateunder the control of the access mechanism store 440, which may specifyparsing routines for various file types.

The instruction data store 436 or the data source itself may specifylocations of metrics within a particular file or data set. As oneexample only, a particular cell of a spreadsheet may contain a metric ofinterest. Therefore, using a parsing routine from the access mechanismstore 440, the data processing module 620 can extract the contents of acell specified by the instruction data store 436 from a spreadsheet fileobtained by the acquisition system 612.

When information about specific locations of data within a document ordata set is encoded within that document or data set itself, theinstruction data store 436 does not need to be updated to reflectchanged locations. This may allow an analyst to, for example, revise aspreadsheet and simply update the spreadsheet accordingly. Continuingthe spreadsheet example, a new location or tab of the spreadsheet maydefine, for each metric contained within that spreadsheet, where themetric can be found. In other implementations, that location or tab ofthe spreadsheet may actually contain a cell that is dynamically updatedwith the value of the metric. For example, the cell may include aformula that obtains the metric from another location in thespreadsheet. In this way, the predetermined location or tab of thespreadsheet will include a copy of the metric value, regardless of wherethat value is produced in the remainder of the spreadsheet. Then, as theanalyst updates the spreadsheet, the analyst simply updates the formulaof the cell to load the metric from its new location.

As a more specific example, the search system may define a specific tabname to be used for publishing metrics of spreadsheets. For aspreadsheet including a metric to be published, this tab is added to thespreadsheet by the analyst or publisher. Each row of this tab maycorresponds to a different metric, with the potential exception of apredefined header row that is ignored by the acquisition system 612.

One column includes a value of the metric, while another column mayspecify units of the metric, which may simply be stored as text forpresentation to users. In no particular order, another column includesthe date or timestamp when the data was last refreshed. The format ofthis column may be predefined and require a specific time zone or aparticular time standard such as Coordinated Universal Time (UTC).Another column specifies the name of the author (such as analyst) of themetric, while yet another column specifies contact information (such asemail address) of the author.

Another column defines the metric in text form, which may be a longerform description, such as one or more full sentences. Another columnspecifies the name of the metric, which may be relatively standardizedacross analysts and may use shorthand for brevity. Another columndefines keywords for the metric, which may conform to a specifiedformat, such as being separated by a predefined delimiter.

Another column defines how frequently the data is refreshed. This callmay be used by the scheduling system 604 to determine when to re-harvestthe metric. In this way, the schedule may be updated anytime the metricis obtained. In other implementations, the refresh cadence column maysimply be used as information to present to users of the search system,while the acquisition schedule is controlled explicitly by the publisheror author via the web server 432.

Another column specifies where the current file can be accessed, such aswith a uniform resource locator (URL). The values in this column may beidentical to the location of the spreadsheet used for access by theharvesting system 444. In other implementations, the harvesting system444 may be accessing the spreadsheet via an alternative path, mounteddrive, etc. Therefore, the source URL may be a more generally applicablelocation accessible by users.

In various implementations, some or all of these columns may be omitted.If a column is omitted, this data may be provided by the publisherexplicitly to the web server 432 or may be omitted from the searchsystem entirely. The only required column is the column including themetric value. As mentioned above, a cell in the metric value column mayinclude a formula that obtains the value from another location orlocations in the spreadsheet. In other implementations, the value columnmay specify where in the spreadsheet the value can be obtained, such asby sheet name, row identifier, and column identifier. In variousimplementations, similar metrics (for example, the same measurement madeover different periods of time, such as a weekly value and a monthlyvalue) are treated as separate metrics and therefore require separaterows.

If data is available in an online source, such as a SharePointcollaboration site from Microsoft Corporation), a list may be definedhaving columns similar to those described for a spreadsheet. This listmay be hidden from view for regular users of the SharePointcollaboration site. The harvesting system 444 may be able to directlyaccess the SharePoint list, such as using a URL pointing to theSharePoint list.

Business intelligence (BI) software (such as Tableau from TableauSoftware) may be configured to output metrics via an API or to a fileusing a predetermined format. For example, the BI software may beconfigured to export metric data to a comma separated value (CSV) fileeach time the underlying data changes or upon request by the harvestingsystem 444. Data similar to the above columns for a spreadsheet may bemanually programmed into the BI software and configured to export alongwith the metric values. As an example only, one approach to adaptingTableau BI software for the harvesting system 444 of the presentdisclosure is described in detail here.

Bisoftware Onboarding Example

In this example, consider a Tableau workbook called “Gym Utilization”that tracks visits to a hypothetical fitness facility. The workbook hasseveral views, including peak hours of use and peak days of use. Themain measure in the workbook is number of visits, with dates and timesas dimensions. In this example, the publisher decides to publish thehours and days of most use and least use.

First, the publisher determines the exact data points to publish. Forexample, these may be: Hour of Most Visits, Hour of Fewest Visits, Dayof Most Visits, Day of Least Visits, and four corresponding VisitCounts. A more complex representation of the data could use a calculatedfield rather than a single value.

Next, the publisher develops names and definitions that are complete andconcise for each data point. For example, these may be:

a. Most popular hour of the day for gym usage in prior month

b. Most number of gym visits per hour in prior month

c. Least popular hour of the day for gym usage in prior month

d. Fewest number of gym visits per hour in prior month

e. Most popular day of the week for gym visits in prior month

f. Average gym visits on most popular day of the week in prior month

g. Least popular day of the week for gym visits in prior month

h. Average gym visits on least popular day of the week in prior month

Next, the publisher identifies the author/analyst and analyst email forinclusion with these data points. As an example, the author may bedesignated as Shun Li, whose email address is shun_li@thepoet.com.

The publisher may then tag the data points with keywords, which may beused for cataloging and searching. The keywords do not need to repeatterms from the metric definitions, which will already be indexed. Forthis example, the keywords will be health, fitness, and exercise.

The publisher then identifies the cadence of the data refresh for thesource of the data point. For example, this gym usage workbook mayupdate on the first Monday of the month at approximately 0400 CentralStandard Time.

To obtain a CSV version of a Tableau view, the publisher opens the viewon the server using a web browser. The publisher then modifies the URLin the address bar by replacing the “?” and any subsequent characterswith “.csv”. The publisher then requests the CSV file by pressing returnor clicking/tapping the browser's “Go” button. The publisher is thenprompted to open or save the CSV file.

The CSV may be sorted by average visits, with row 1 being a header rowand fourteen data points (rows 2-15). As an example, assume that thesecond column contains the average visit numbers. Therefore, the datapoints for greatest and fewest visit can be obtained from rows 2 and 15of column 2 of the CSV.

However, the publisher should consider whether a different sort ordermay apply that will change the row for the data point. Further, thepublisher should consider whether the data in question always appears inthe view, such as when the value is <NULL>. Further, the publishershould consider whether the data relies on filter settings or parametersthat could change due to data changes. If the location of the datapoints may change in the output file, the publisher might duplicate theview in a separate tab of the Tableau view and make appropriateadjustments to filters or view construction. As another option, thepublisher may programmatically calculate the row number each time thedata is updated. This row number would then need to be acquired by theharvesting system to determine from which row to obtain the metricvalue.

To prepare the Tableau workbook for harvesting, the publisher opens theworkbook in the desktop Tableau application and navigates to the viewwith the data. For this example, the search system is named FAST(Frequently Asked STatistics) and therefore predefined names will beprefaced with “FAST”. The publisher creates calculated fields with names“Author”, “Author Email”, “FASTMetric”, “FASTTags”, and “RefreshCadence”. In various implementations, the field names arecase-sensitive.

The publisher then edits the Author and Author Email fields to reflectthe author and/or subject matter expert for this data. The publisheredits the Refresh Cadence field to reflect the schedule of data refresh.The publisher edits the FASTTags field to insert terms delimited withsemicolons and no other delimiters, such as spaces or commas.

The publisher edits the FASTMetric field to reflect the location of thedata point in the CSV grid and a description of the data point. Each rowof the FASTMetric field corresponds to one metric to be published. Eachitem in the row is separated by a semicolon. The FASTMetric field can beconstructed as follows:

-   a. Type a double quote (“).-   b. Type or copy/paste the name of the column header of the data from    the CSV file exactly as it is displayed in the CSV file, followed by    a semicolon.-   c. Type the number of the row for the data (row 1 contains the    column headers) followed by a semicolon.-   d. Type the desired Display Name for the data point (this may be the    same as the column header) followed by a semicolon.-   e. Type the Definition/Comments/Other information about the data    followed by a semicolon. This provides context for the data as it    will appear by itself in a search result.-   f. Avoid special characters such as “{ }\; unless specifically    instructed.-   g. Hit <Return>.-   h. Repeat steps b through g for each data point to harvest. Each    data point can have different columns and rows.-   i. Type a closing double quote (”).

When constructed as above for the four metrics in the current Tableauview, the FASTMetric field is as follows:

“Hour of In;2;Gym Visit Start Time;Most popular hour of day for Gymusage; Avg Visits;2;Number of Gym Visits per hour;Most Number of Visitsper hour; Hour of In;15;Gym Visit Start Time;Least popular hour of dayfor Gym usage; Avg Visits;15;Number of Gym Visits per hour;Least Numberof visits per hour”

Note that the first entry is the column header from the CSV file thatcontains the data point. The next entry is the row number (2 or 15). Thenext entry is a display name for the data point, and the last entry isthe description. The four lines in the FASTMetric field correspond tothe four separate data points. Add each of the created fields to thedetails card of the Tableau view. They can be eliminated from thetooltip without affecting harvesting. This view is then ready forharvesting.

Consider another view for this fitness facility workbook. This viewcontains average number of gym visits by day of the week and cantherefore be used to determine the most and least popular days of theweek for gym visits in the prior month. The CSV output file may includea column for day of the week and a column for average number of gymvisits, with the data sorted by day of the week. As a result, the rowthat contains the most visits is not always the same, and neither is therow that contains the fewest visits.

To address this issue, the publisher creates a duplicate view andchanges the sort order to number of visits instead of days of the week.For example, the duplicate worksheet can be named FAST_DayOfWeek. Thesort order in the FAST_DayOfWeek worksheet is set to number of visits indescending order. Then, the CSV output file for the FAST_DayOfWeekworksheet will (assuming visits are tracked from Monday-Friday and noton the weekends), include a header row and five weekday rows, with thegreatest number of visits being in row 2 and the fewest number of visitsbeing in row 6.

The FASTMetric field (this one may be titled FASTMetric2 todifferentiate from the FASTMetric field in the other view) can beconstructed as follows:

“Day of the Week;2;Day of the Week;Most popular day of the week for GymVisits

Avg Visits (copy);2;Average Gym Visits;Number of Average Visits to theGym on most popular day of the week

Day of the Week;6;Day of the Week;Least popular day of the week for GymVisits

Avg Visits (copy);6;Average Gym Visits;Number of Average Visits to theGym on least popular day of the week”

Now the workbook with the FAST_DayOfWeek worksheet is published and theCSV URL is provided to the web server 432. The harvesting system canthen review FASTMetric* fields from the URL to determine how to obtainmetric values.

Some idiosyncrasies to keep in mind are that Tableau exports CSV fileswith column headings sorted left to right with an alphabetical sort ofDimensions and Measures that are created within Tableau Desktop, such asCalculated Fields, Groups, and Sets. Then, Tableau lists the remainingDimensions and Measures from the data source in alphabetical order.

Calculated fields are used to perform row sort before any otherdimensions and only are sorted on their value (even if the sort is setto another field, this won't be reflected in the CSV). This property maybe used to create a specific row sort order. For example, make thecalculated field value represent the desired sort order and label itstarting with an underscore: for example, as in “_Counts”. This willmake sure it is first in the column list and therefore be the sort ofthe CSV file.

Flowcharts

In FIG. 7, example operation of an administrative interface (such aspresented by the web server 452) for publishers and analysts begins at704. If a login request is made by a user, control transfers to 708;otherwise, control remains at 704. At 708, control attempts toauthenticate the user. At 712, if authentication was successful, controltransfers to 716; otherwise, control returns to 704. For example only,authentication may be based on an Active Directory identity system fromMicrosoft Corporation or may rely on tokens generated by asingle-sign-on process.

At 716, if an existing metric is selected by a user, control transfersto 720; otherwise, control transfers to 724. At 720, control displaysjust statistics regarding the metric. At 728, control determines whetherchanges have been requested by the user. If so, control transfers to732; otherwise, control transfers to 724. At 732, control updates theinstruction data store based on the user input. For example, thelocation of the metric may have changed and therefore the user providesa new URL.

At 724, control determines whether the user has indicated an intent topublish a new metric. If so, control transfers to 736; otherwise,control transfers to 740. At 740, control determines whether the loginis still active. If so, control transfers to 716; otherwise, controlreturns to 704. At 736, control obtains metadata for the metric,including the source (such as the URL) and an update schedule. Controlmay also acquire additional metadata regarding the metric.

At 744, control attempts to acquire the metric based on the specifiedsource. At 748, if acquisition was successful, control transfers to 752;otherwise, control returns to 736 to request corrected metadata. At 752,control displays the acquired metric to the user for confirmation. At756, control determines whether the user has instructed enabling themetric to be displayed. For example, the user may desire to have thesearch engine begin acquiring the metric before displaying the metric tousers. If the user decides to enable the display, control transfers to760; otherwise, control transfers to 740.

At 760, in the case an administrator is required to approve new metrics,the administrator may review the metric provided by the user. Althoughshown immediately after 756, administrator review may happenasynchronously, such as in response to an alert of a new metric, ordaily. At 760, if the administrator enables display of the metric,control transfers to 764; otherwise, control transfers to 740. At 764,control sets the new metric as visible in search results, allowing themetric to be presented to users in response to queries. Control thenreturns to 740.

In FIG. 8, example operation of the harvesting system 444 begins at 804.For example, control may begin on a periodic schedule, such as once persecond. At 804, control reads instructions, including schedules, fromthe instruction data store 436. At 808, control selects the first metricsource. At 812, control identifies all of the refresh schedulespertinent to the selected document source and selects the first of thoserefresh schedules.

At 816, control determines whether it is time to refresh any data fromthe selected source having the selected schedule. For example, if anymetric was obtained prior to the most recent event in the selectedrefresh schedule, that metric is eligible for reacquisition. If anymetric is potentially out of date based on the selected refreshschedule, control transfers to 820; otherwise, control transfers to 824.At 824, control determines whether there are any additional schedulespertinent to the selected data source. If so, control transfers to 828;otherwise, control transfers to 832. At 828, control selects the nextrefresh schedule applicable to the selected metric source and continuesat 816. At 832, control determines whether there are any additionalmetric sources specified by the instruction data store 436. If so,control transfers to 836; otherwise, control ends. At 836, controlselects the next metric source and continues at 812.

At 820, control selects an access mechanism based on the type of theselected metric source. These access mechanisms may be maintained by theaccess mechanism store 440. For example, the access mechanism may permitsubmission of a SQL query or access to a SharePoint collaboration sitelist object. The access mechanism store 440 may also specify how toobtain modification date information. At 840, if the access mechanismstore 440 indicates that modification date information is potentiallyavailable, control transfers to 844; otherwise, control transfers to848. At 844, control obtains the modification date for the metric usingthe selected access mechanism. Control then continues at 852. If themost recent value of any metric is older than the update date, controltransfers to 848; otherwise, control transfers to 824.

At 848, using the selected access mechanism, control attempts toretrieve values and metadata for metrics sharing the selected refreshschedule from the selected metric source. In response to an error, suchas predefined names not being present in the selected metric source,control may signal an error to an administrator of the search system andreturn to 824.

At 856, control may execute instructions indicated by the metadata, suchas accessing a particular cell within the spreadsheet. As an example, apredefined location within a spreadsheet may include a row for eachmetric, where that row indicates a location of the metric valuethroughout the spreadsheet. At 860, control stores the obtained metricdata into the metric data store 456. Control then continues at 824.

CONCLUSION

The foregoing description is merely illustrative in nature and is in noway intended to limit the disclosure, its application, or uses. Thebroad teachings of the disclosure can be implemented in a variety offorms. Therefore, while this disclosure includes particular examples,the true scope of the disclosure should not be so limited since othermodifications will become apparent upon a study of the drawings, thespecification, and the following claims. It should be understood thatone or more steps within a method may be executed in different order (orconcurrently) without altering the principles of the present disclosure.Further, although each of the embodiments is described above as havingcertain features, any one or more of those features described withrespect to any embodiment of the disclosure can be implemented in and/orcombined with features of any of the other embodiments, even if thatcombination is not explicitly described. In other words, the describedembodiments are not mutually exclusive, and permutations of one or moreembodiments with one another remain within the scope of this disclosure.

Spatial and functional relationships between elements (for example,between modules) are described using various terms, including“connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitlydescribed as being “direct,” when a relationship between first andsecond elements is described in the above disclosure, that relationshipencompasses a direct relationship where no other intervening elementsare present between the first and second elements, and also an indirectrelationship where one or more intervening elements are present (eitherspatially or functionally) between the first and second elements. Asused herein, the phrase at least one of A, B, and C should be construedto mean a logical (A OR B OR C), using a non-exclusive logical OR, andshould not be construed to mean “at least one of A, at least one of B,and at least one of C.”

In the figures, the direction of an arrow, as indicated by thearrowhead, generally demonstrates the flow of information (such as dataor instructions) that is of interest to the illustration. For example,when element A and element B exchange a variety of information butinformation transmitted from element A to element B is relevant to theillustration, the arrow may point from element A to element B. Thisunidirectional arrow does not imply that no other information istransmitted from element B to element A. Further, for information sentfrom element A to element B, element B may send requests for, or receiptacknowledgements of, the information to element A. The term subset doesnot necessarily require a proper subset. In other words, a first subsetof a first set may be coextensive with (equal to) the first set.

In this application, including the definitions below, the term “module”or the term “controller” may be replaced with the term “circuit.” Theterm “module” may refer to, be part of, or include processor hardware(shared, dedicated, or group) that executes code and memory hardware(shared, dedicated, or group) that stores code executed by the processorhardware.

The module may include one or more interface circuits. In some examples,the interface circuit(s) may implement wired or wireless interfaces thatconnect to a local area network (LAN) or a wireless personal areanetwork (WPAN). Examples of a LAN are Institute of Electrical andElectronics Engineers (IEEE) Standard 802.11-2016 (also known as theWIFI wireless networking standard) and IEEE Standard 802.3-2015 (alsoknown as the ETHERNET wired networking standard). Examples of a WPAN arethe BLUETOOTH wireless networking standard from the Bluetooth SpecialInterest Group and IEEE Standard 802.15.4.

The module may communicate with other modules using the interfacecircuit(s). Although the module may be depicted in the presentdisclosure as logically communicating directly with other modules, invarious implementations the module may actually communicate via acommunications system. The communications system includes physicaland/or virtual networking equipment such as hubs, switches, routers, andgateways. In some implementations, the communications system connects toor traverses a wide area network (WAN) such as the Internet. Forexample, the communications system may include multiple LANs connectedto each other over the Internet or point-to-point leased lines usingtechnologies including Multiprotocol Label Switching (MPLS) and virtualprivate networks (VPNs).

In various implementations, the functionality of the module may bedistributed among multiple modules that are connected via thecommunications system. For example, multiple modules may implement thesame functionality distributed by a load balancing system. In a furtherexample, the functionality of the module may be split between a server(also known as remote, or cloud) module and a client (or, user) module.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. Shared processor hardware encompasses asingle microprocessor that executes some or all code from multiplemodules. Group processor hardware encompasses a microprocessor that, incombination with additional microprocessors, executes some or all codefrom one or more modules. References to multiple microprocessorsencompass multiple microprocessors on discrete dies, multiplemicroprocessors on a single die, multiple cores of a singlemicroprocessor, multiple threads of a single microprocessor, or acombination of the above.

Shared memory hardware encompasses a single memory device that storessome or all code from multiple modules. Group memory hardwareencompasses a memory device that, in combination with other memorydevices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium is therefore considered tangible and non-transitory. Non-limitingexamples of a non-transitory computer-readable medium are nonvolatilememory devices (such as a flash memory device, an erasable programmableread-only memory device, or a mask read-only memory device), volatilememory devices (such as a static random access memory device or adynamic random access memory device), magnetic storage media (such as ananalog or digital magnetic tape or a hard disk drive), and opticalstorage media (such as a CD, a DVD, or a Blu-ray Disc).

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks andflowchart elements described above serve as software specifications,which can be translated into the computer programs by the routine workof a skilled technician or programmer.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium. Thecomputer programs may also include or rely on stored data. The computerprograms may encompass a basic input/output system (BIOS) that interactswith hardware of the special purpose computer, device drivers thatinteract with particular devices of the special purpose computer, one ormore operating systems, user applications, background services,background applications, etc.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language), XML (extensible markuplanguage), or JSON (JavaScript Object Notation), (ii) assembly code,(iii) object code generated from source code by a compiler, (iv) sourcecode for execution by an interpreter, (v) source code for compilationand execution by a just-in-time compiler, etc. As examples only, sourcecode may be written using syntax from languages including C, C++, C#,Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl,Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5threvision), Ada, ASP (Active Server Pages), PHP (PHP: HypertextPreprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, VisualBasic®, Lua, MATLAB, SIMULINK, and Python®.

What is claimed is:
 1. An administrative interface configured to be usedwith one or more databases for analytics comprising: specifying, via theadministrative interface, a first predefined label for use with the oneor more databases for analytics; receiving a first input designating afirst location of a first analytic; obtaining a first document from thefirst location; identifying the first predefined label within the firstdocument; obtaining a first datum associated with the first predefinedlabel from the first document; storing the first datum into a valueindex as a current value of the first analytic; obtaining a second datumassociated with the first predefined label from the first document;storing the second datum into a text index as a textual description ofthe first analytic; and determining a level of authority to associatewith the first analytic.
 2. The administrative interface of claim 1,wherein determining a level of authority to associate with the firstanalytic comprises: in response to obtaining the first document,acquiring, from a predefined position in the first document with respectto the first predefined label, identification information for one ormore of an author and a publisher of the first analytic; acquiringcredibility data associated with the one or more of the author and thepublisher of the first analytic; determining a credibility value toassociate with the first analytic based, at least in part, on theacquired credibility data; and utilizing the credibility value, at leastin part, to determine the level of authority to associate with the firstanalytic.
 3. The administrative interface of claim 2, furthercomprising: selectively displaying the identification information forone or more of the author and a publisher of the first analytic to auser.
 4. The administrative interface of claim 3, wherein selectivelydisplaying the identification information comprises: acquiring, from asecond predefined position in the first document with respect to thefirst predefined label, contact information for one or more of theauthor and the publisher; and selectively displaying the contactinformation to the user.
 5. The administrative interface of claim 1,wherein determining a level of authority to associate with the firstanalytic comprises: assigning, by a publisher of the first analytic, aconfidence score to the first analytic; and utilizing the confidencescore, at least in part, to determine the level of authority toassociate with the first analytic.
 6. The administrative interface ofclaim 5, further comprising: invalidating the confidence scoreassociated with the first analytic based upon expiration of apredetermined amount of time.
 7. The administrative interface of claim1, wherein determining a level of authority to associate with the firstanalytic comprises: receiving, from one or more users of the searchmethod, user information indicating whether the one or more usersdetermined the first analytic to be authoritative or helpful; andutilizing the user information, at least in part, to determine the levelof authority to associate with the first analytic.
 8. A systemconfigured to be used with one or more databases for analyticscomprising: at least one processor; and a computer-readable mediumconfigured to store instructions for execution by the at least oneprocessor, wherein the instructions include: specifying a firstpredefined label for use with the system; receiving first inputdesignating a first location of a first analytic; obtaining a firstdocument from the first location; identifying the first predefined labelwithin the first document; obtaining a first datum associated with thefirst predefined label from the first document; storing the first datuminto a value index as a current value of the first analytic; obtaining asecond datum associated with the first predefined label from the firstdocument; storing the second datum into a text index as a textualdescription of the first analytic; and determining a level of authorityto associate with the first analytic.
 9. The system of claim 8, whereindetermining a level of authority to associate with the first analyticcomprises: in response to obtaining the first document, acquiring, froma predefined position in the first document with respect to the firstpredefined label, identification information for one or more of anauthor and a publisher of the first analytic; acquiring credibility dataassociated with the one or more of the author and the publisher of thefirst analytic; determining a credibility value to associate with thefirst analytic based, at least in part, on the acquired credibilitydata; and utilizing the credibility value, at least in part, todetermine the level of authority to associate with the first analytic.10. The system of claim 9, further comprising: selectively displayingthe identification information for one or more of the author and apublisher of the first analytic to a user.
 11. The system of claim 10,wherein selectively displaying the identification information comprises:acquiring, from a second predefined position in the first document withrespect to the first predefined label, contact information for one ormore of the author and the publisher; and selectively displaying thecontact information to the user.
 12. The system of claim 8, whereindetermining a level of authority to associate with the first analyticcomprises: assigning, by a publisher of the first analytic, a confidencescore to the first analytic; and utilizing the confidence score, atleast in part, to determine the level of authority to associate with thefirst analytic.
 13. The system of claim 12, further comprising:invalidating the confidence score associated with the first analyticbased upon expiration of a predetermined amount of time.
 14. The systemof claim 8, wherein determining a level of authority to associate withthe first analytic comprises: receiving, from one or more users of thesearch method, user information indicating whether the one or more usersdetermined the first analytic to be authoritative or helpful; andutilizing the user information, at least in part, to determine the levelof authority to associate with the first analytic.
 15. A non-transitorycomputer-readable medium storing processor-executable instructions, theinstructions comprising: specifying a first predefined label, whereinspecifying the first predefined label includes using an administrativeinterface configured to be used with one or more databases foranalytics; receiving first input designating a first location of a firstanalytic; obtaining a first document from the first location;identifying the first predefined label within the first document;obtaining a first datum associated with the first predefined label fromthe first document; storing the first datum into a value index as acurrent value of the first analytic; obtaining a second datum associatedwith the first predefined label from the first document; storing thesecond datum into a text index as a textual description of the firstanalytic; and determining a level of authority to associate with thefirst analytic.
 16. The non-transitory computer-readable medium storingprocessor-executable instructions of claim 15, wherein determining alevel of authority to associate with the first analytic comprises: inresponse to obtaining the first document, acquiring, from a predefinedposition in the first document with respect to the first predefinedlabel, identification information for one or more of an author and apublisher of the first analytic; acquiring credibility data associatedwith the one or more of the author and the publisher of the firstanalytic; determining a credibility value to associate with the firstanalytic based, at least in part, on the acquired credibility data; andutilizing the credibility value, at least in part, to determine thelevel of authority to associate with the first analytic.
 17. Thenon-transitory computer-readable medium storing processor-executableinstructions of claim 16, further comprising: selectively displaying theidentification information for one or more of the author and a publisherof the first analytic to a user; acquiring, from a second predefinedposition in the first document with respect to the first predefinedlabel, contact information for one or more of the author and thepublisher; and selectively displaying the contact information to theuser.
 18. The non-transitory computer-readable medium storingprocessor-executable instructions of claim 15, wherein determining alevel of authority to associate with the first analytic comprises:assigning, by a publisher of the first analytic, a confidence score tothe first analytic; and utilizing the confidence score, at least inpart, to determine the level of authority to associate with the firstanalytic.
 19. The non-transitory computer-readable medium storingprocessor-executable instructions of claim 18, further comprising:invalidating the confidence score associated with the first analyticbased upon expiration of a predetermined amount of time.
 20. Thenon-transitory computer-readable medium storing processor-executableinstructions of claim 15, wherein determining a level of authority toassociate with the first analytic comprises: receiving, from one or moreusers of the search method, user information indicating whether the oneor more users determined the first analytic to be authoritative orhelpful; and utilizing the user information, at least in part, todetermine the level of authority to associate with the first analytic.